Measuring Health Care Work-Related Contextual Factors: Development of the McGill Context Tool.

INTRODUCTION: Contextual factors can influence healthcare professionals' (HCPs) competencies, yet there is a scarcity of research on how to optimally measure these factors. The aim of this study was to develop and validate a comprehensive tool for HCPs to document the contextual factors likely to influence the maintenance, development, and deployment of professional competencies. METHODS: We used DeVellis' 8-step process for scale development and Messick's unified theory of validity to inform the development and validation of the context tool. Building on results from a scoping review, we generated an item pool of contextual factors articulated around five themes: Leadership and Agency, Values, Policies, Supports, and Demands. A first version of the tool was pilot tested with 127 HCPs and analyzed using the classical test theory. A second version was tested on a larger sample (n = 581) and analyzed using the Rasch rating scale model. RESULTS: First version of the tool: we piloted 117 items that were grouped as per the themes related to contextual factors and rated on a 5-point Likert scale. Cronbach alpha for the set of 12 retained items per scale ranged from 0.75 to 0.94. Second version of the tool included 60 items: Rasch analysis showed that four of the five scales (ie, Leadership and Agency, Values, Policies, Supports) can be used as unidimensional scales, whereas the fifth scale (Demands) had to be split into two unidimensional scales (Demands and Overdemands). DISCUSSION: Validity evidence documented for content and internal structure is encouraging and supports the use of the McGill context tool. Future research will provide additional validity evidence and cross-cultural translation.

An exploration of values in medical school admissions processes: the interplay between contextual factors, admissions practices, and validity.

Medical school admissions is a contentious and high stakes selection activity. Many assessment approaches are available to support selection; but how are decisions about building, monitoring, and adapting admissions systems made? What shapes the processes and practices that underpin selection decisions? We explore how these decisions are made across several Canadian medical schools, and how values shape the creation, monitoring, and adaptation of admissions systems. Using phenomenography (a qualitative method suited to examining variability), the authors analyzed interviews with 10 current or previous heads of admissions from 10 different undergraduate medical education programs in Canada. Interviews were conducted in English and French, and data was collected between 2016 and 2017 (therefore participants no longer hold these roles). Data was coded and analyzed iteratively, focusing on identifying underlying values, and exploring how these values shape admissions practices and considerations for validity. Eight different intersecting values were identified. Of these, four were shared across all participants: critically questioning the process and tools, aiming for equity, striving for better, and embracing the challenges of change. The expression of these values depended on different contextual variables (e.g., geographic location, access to expertise, resource availability), and values shaped how admissions systems were built, enacted, and monitored for quality. Ultimately, values shaped: (1) admissions practices resulting in different candidates being offered admission, and (2) how arguments supporting score interpretation are built (i.e., validity). This study documents various values that influence admissions processes, practices, and quality monitoring. The values that shape what is assessed, how it is assessed, and how fair and defensible practices are conceptualized have significant impact, ultimately determining who is selected. These values-whether implicit or explicit-result in intended and unintended consequences in selection processes. However, these values are rarely explicitly examined and questioned, leaving it uncertain as to which consequences are the intended outcomes of deliberately chosen values, and which are unintended consequences of implicitly held values of admissions systems and their actors.

Exploring medical students' use of principles of self-explanation and structured reflection during clerkship.

Background: While educators observe gaps in clerkship students' clinical reasoning (CR) skills, students report few opportunities to develop them. This study aims at exploring how students who used self-explanation (SE) and structured reflection (SR) for CR learning during preclinical training, applied these learning strategies during clerkship. Methods: We conducted an explanatory sequential mixed-methods study involving medical students. With a questionnaire, we asked students how frequently they adopted behaviours related to SE and SR during clerkship. Next, we conducted a focus group with students to explore why they adopted these behaviours. Results: Fifty-two of 198 students answered the questionnaire and five participated in a focus group. Specific behaviours adopted varied from 50% to 98%. We identified three themes about why students used these strategies: as "just in time" learning strategies; to deepen their understanding and identify gaps in knowledge; to develop a practical approach to diagnosis. A fourth theme related to the balance between learning and assessment and its consequence on adopting SE behaviours. Conclusions: Students having experienced SE and SR regularly in preclinical training tend to transpose these strategies into the clerkship providing them with a practical way to reflect deliberately and capture learning opportunities of the unpredictable clinical context.

Contexte: Alors que les éducateurs observent des lacunes dans les compétences de raisonnement clinique (RC) des étudiants en externat, ces derniers font état de peu d'occasions de les développer. Cette étude vise à explorer comment les étudiants qui ont utilisé l'auto-explication (AE) et la réflexion structurée (RS) pour l'apprentissage du raisonnement clinique pendant la formation préclinique, ont appliqué ces stratégies d'apprentissage pendant l'externat. Méthodes: Nous avons mené une étude séquentielle explicative à méthodes mixtes auprès d'étudiants en médecine. À l'aide d'un questionnaire, nous avons demandé aux étudiants à quelle fréquence ils adoptaient des comportements liés à la AE et à la RS pendant l'externat. Ensuite, nous avons organisé un groupe de discussion avec les étudiants afin d'explorer les raisons pour lesquelles ils ont adopté ces comportements. Résultats: Cinquante-deux étudiants sur 198 ont répondu au questionnaire et cinq ont participé à un groupe de discussion. Les comportements spécifiques adoptés variaient de 50 % à 98 %. Nous avons identifié trois thèmes concernant les raisons pour lesquelles les étudiants ont utilisé ces stratégies : comme stratégies d'apprentissage "juste à temps"; pour approfondir leur compréhension et identifier les lacunes dans les connaissances ; pour développer une approche pratique du diagnostic. Un quatrième thème concernait l'équilibre entre l'apprentissage et l'évaluation et ses conséquences sur l'adoption de comportements liés à l'AE. Conclusions: Les étudiants qui ont fait l'expérience de l'AE et de la RS régulièrement au cours de leur formation préclinique ont tendance à transposer ces stratégies dans l'externat, ce qui leur fournit un moyen pratique de réfléchir délibérément et de saisir les opportunités d'apprentissage dans un contexte clinique imprévisible.

Assessing commitment to reflection: perceptions of medical students.

Background: While developing reflection skills is considered important by educators, the assessment of these skills is often associated with unintended negative consequences. In the context of a mandatory longitudinal course that aims to promote the development of reflection on professional identity, we assessed students' commitment to reflection. This study explores students' perception of this assessment by their mentor. Methods: We conducted a qualitative descriptive study using semi-structured interviews with twenty-one 1st and six 2nd year medical students. Thematic analysis was informed by Braun and Clarke's six-step approach. Results: We identified four main themes: 1- assessment as a motivator, 2- consequences on authenticity, 3- perception of inherent subjectivity, and 4 - relationship with the mentor. Conclusions: In the context of assessing reflection skills in future physicians, we observed that students -when assessed on the process of reflection- experienced high motivation but were ambivalent on the question of authenticity. The subjectivity of the assessment as well as the relationship with their mentor also raises questions. Nevertheless, this assessment approach for reflective skills appears to be promising in terms of limiting the negative consequences of assessment.

Contexte: Malgré l'importance que les éducateurs attribuent à l'acquisition de compétences de réflexion, l'évaluation de ces compétences entraîne souvent des conséquences négatives involontaires. Dans le cadre d'un cours longitudinal obligatoire visant à promouvoir le développement de la réflexion sur l'identité professionnelle, nous avons évalué l'engagement des étudiants à cultiver leurs compétences de réflexion. Cette étude explore leur perception de cette évaluation menée par leur mentor. Méthodes: Nous avons réalisé une étude qualitative descriptive à l'aide d'entretiens semi-structurés avec vingt-et-un étudiants en médecine de première année et six étudiants en médecine de deuxième année. Notre analyse thématique repose sur l'approche en six étapes de Braun et Clarke. Résultats: Nous avons identifié quatre thèmes principaux : 1 ­ l'évaluation comme facteur de motivation, 2 ­ les conséquences sur l'authenticité, 3 ­ la perception de la subjectivité inhérente, et 4 ­ la relation avec le mentor. Conclusions: Dans le contexte de l'évaluation des compétences de réflexion des futurs médecins, focalisée plus particulièrement sur le processus de réflexion, les étudiants se sont montrés très motivés, mais incertains quant à son authenticité. La subjectivité de l'évaluation et la relation avec leur mentor soulèvent également des interrogations. Néanmoins, cette approche d'évaluation des compétences réflexives semble prometteuse dans la mesure où elle permet de limiter les conséquences négatives de l'évaluation.

Assessment Practices in Continuing Professional Development Activities in Health Professions: A Scoping Review.

CONTEXT: In continuing professional development (CPD), educators face the need to develop and implement innovative assessment strategies to adhere to accreditation standards and support lifelong learning. However, little is known about the development and validation of these assessment practices. We aimed to document the breadth and depth of what is known about the development and implementation of assessment practices within CPD activities. METHODS: We conducted a scoping review using the framework proposed by Arksey and O'Malley (2005) and updated in 2020. We examined five databases and identified 1733 abstracts. Two team members screened titles and abstracts for inclusion/exclusion. After data extraction, we conducted a descriptive analysis of quantitative data and a thematic analysis of qualitative data. RESULTS: A total of 130 studies were retained for the full review. Most reported assessments are written assessments (n = 100), such as multiple-choice items (n = 79). In 99 studies, authors developed an assessment for research purpose rather than for the CPD activity itself. The assessment validation process was detailed in 105 articles. In most cases, the authors examined the content with experts (n = 57) or pilot-tested the assessment (n = 50). We identified three themes: 1-satisfaction with assessment choices; 2-difficulties experienced during the administration of the assessment; and 3-complexity of the validation process. CONCLUSION: Building on the adage "assessment drives learning," it is imperative that the CPD practices contribute to the intended learning and limit the unintended negative consequences of assessment. Our results suggest that validation processes must be considered and adapted within CPD contexts.

Considering the Structured Oral Examinations Beyond Its Psychometrics Properties.

Decisions to set aside Structured Oral Examinations (SOE) are, almost invariably, based on their poor psychometric properties. However, considering the perspectives of the stakeholders might help us to understand its potential contribution. To explore this, we conducted focus groups and individual interviews with stakeholders: students, assessors, and administrators. Students and assessors perceived the SOE as a window on students' clinical reasoning, as an authentic assessment, but as a subjective and stressful method. Administrators emphasized the organizational consequences such as logistical challenges. Consequences must be considered when making decisions about SOE and our results support important positive consequences.

Assessing trainee's performance using reported observations: Perceptions of nurse meta-assessors.

BACKGROUND: Educational and health care organizations who prepare meta-assessors to fulfill their role in the assessment of trainees' performance based on reported observations have little literature to rely on. While the assessment of trainees' performance based on reported observations has been operationalized, we have yet to understand the elements that can affect its quality fully. Closing this gap in the literature will provide valuable insight that could inform the implementation and quality monitoring of the assessment of trainees' performance based on reported observations. OBJECTIVES: The purpose of this study was to explore the elements to consider in the assessment of trainees' performance based on reported observations from the perspectives of meta-assessors. METHODS: Design, Settings, Participants, data collection and analysis. The authors adopted Sandelowski's qualitative descriptive approach to interview nurse meta-assessors from two nursing programs. A semi-structured interview guide was used to document the elements to consider in the assessment of nursing trainees' performance based on reported observations, and a survey was used to collect sociodemographic data. The authors conducted a thematic analysis of the interview transcripts. RESULTS: Thirteen meta-assessors participated in the study. Three core themes were identified: (1) meta-assessors' appropriation of their perceived assessment roles and activities, (2) team climate of information sharing, and (3) challenges associated with the assessment of trainees' performance based on reported observations. Each theme is comprised of several sub themes. CONCLUSIONS: To optimize the quality of the assessment of the trainee's performance based on reported observations and ratings, HPE programs might consider how to clarify better the meta-assessor's roles and activities, as well as how interventions could be created to promote a climate of information sharing and to address the challenges identified. This work will guide educational and health care organizations for better preparation and support for meta-assessors and preceptors.

Quality of Narratives in Assessment: Piloting a List of Evidence-Based Quality Indicators.

Background & Need for Innovation: Appraising the quality of narratives used in assessment is challenging for educators and administrators. Although some quality indicators for writing narratives exist in the literature, they remain context specific and not always sufficiently operational to be easily used. Creating a tool that gathers applicable quality indicators and ensuring its standardized use would equip assessors to appraise the quality of narratives. Steps taken for Development and Implementation of innovation: We used DeVellis' framework to develop a checklist of evidence-informed indicators for quality narratives. Two team members independently piloted the checklist using four series of narratives coming from three different sources. After each series, team members documented their agreement and achieved a consensus. We calculated frequencies of occurrence for each quality indicator as well as the interrater agreement to assess the standardized application of the checklist. Outcomes of Innovation: We identified seven quality indicators and applied them on narratives. Frequencies of quality indicators ranged from 0% to 100%. Interrater agreement ranged from 88.7% to 100% for the four series. Critical Reflection: Although we were able to achieve a standardized application of a list of quality indicators for narratives used in health sciences education, it does not exclude the fact that users would need training to be able to write good quality narratives. We also noted that some quality indicators were less frequent than others and we suggested a few reflections on this.

The Definitions and Conceptualizations of the Practice Context in the Health Professions: A Scoping Review.

INTRODUCTION: Health care professionals work in different contexts, which can influence professional competencies. Despite existing literature on the impact of context on practice, the nature and influence of contextual characteristics, and how context is defined and measured, remain poorly understood. The aim of this study was to map the breadth and depth of the literature on how context is defined and measured and the contextual characteristics that may influence professional competencies. METHODS: A scoping review using Arksey and O'Malley's framework. We searched MEDLINE (Ovid) and CINAHL (EBSCO). Our inclusion criteria were studies that reported on context or relationships between contextual characteristics and professional competencies or that measured context. We extracted data on context definitions, context measures and their psychometric properties, and contextual characteristics influencing professional competencies. We performed numerical and qualitative analyses. RESULTS: After duplicate removal, 9106 citations were screened and 283 were retained. We compiled a list of 67 context definitions and 112 available measures, with or without psychometric properties. We identified 60 contextual factors and organized them into five themes: Leadership and Agency, Values, Policies, Supports, and Demands. DISCUSSION: Context is a complex construct that covers a wide array of dimensions. Measures are available, but none include the five dimensions in one single measure or focus on items targeting the likelihood of context influencing several competencies. Given that the practice context plays a critical role in health care professionals' competencies, stakeholders from all sectors (education, practice, and policy) should work together to address those contextual characteristics that can adversely influence practice.

Will ChatGPT's Free Language Editing Service Level the Playing Field in Science Communication?: Insights from a Collaborative Project with Non-native English Scholars.

ChatGPT has been widely heralded as a way to level the playing field in scientific communication through its free language editing service. However, such claims lack systematic evidence. A writing scholar (LL) and six non-native English scholars researching health professions education collaborated on this Writer's Craft to fill this gap. Our overarching aim was to provide experiential evidence about ChatGPT's performance as a language editor and writing coach. We implemented three cycles of a systematic procedure, describing how we developed our prompts, selected text for editing, incrementally prompted to refine ChatGPT's responses, and analyzed the quality of its language edits and explanations. From this experience, we offer five insights, and we conclude that the optimism about ChatGPT's capacity to level the playing field for non-native English writers should be tempered. In the writer's craft section we offer simple tips to improve your writing in one of three areas: Energy, Clarity and Persuasiveness. Each entry focuses on a key writing feature or strategy, illustrates how it commonly goes wrong, teaches the grammatical underpinnings necessary to understand it and offers suggestions to wield it effectively. We encourage readers to share comments on or suggestions for this section on Twitter, using the hashtag: #how'syourwriting?

Harnessing a knowledge translation framework to implement an undergraduate medical education intervention: A longitudinal study.

INTRODUCTION: Implementation of evidence-informed educational interventions (EEI) involves applying and adapting theoretical and scientific knowledge to a specific context. Knowledge translation (KT) approaches can both facilitate and structure the process. The purpose of this paper is to describe lessons learned from applying a KT approach to help implement an EEI for clinical reasoning in medical students. METHODS: Using the Knowledge to Action framework, we designed and implemented an EEI intended to support the development of students' clinical reasoning skills in a renewed medical curriculum. Using mixed-methods design, we monitored students' engagement with the EEI longitudinally through a platform log; we conducted focus groups with students and stakeholders, and observed the unfolding of the implementation and its continuation. Data are reported according to six implementation outcomes: Fidelity, Feasibility, Appropriateness, Acceptability, Adoption, and Penetration. RESULTS: Students spent a mean of 24 min on the activity (fidelity outcome) with a high completion rate (between 75% and 95%; feasibility outcome) of the entire activity each time it was done. Focus group data from students and stakeholders suggest that the activity was acceptable, appropriate, feasible, adopted and well-integrated into the curriculum. DISCUSSION: Through the process we observed the importance of having a structuring framework, of working closely and deliberatively with stakeholders and students, of building upon concurrent evaluations in order to adapt iteratively the EEI to the local context and, while taking students' needs into consideration, of upholding the EEI's core educational principles.

Canadian general internal medicine residents' perception of a pedagogical tool of online cases in obstetric medicine.

Background: Sufficient exposure to rarer medical problems around pregnancy is a challenge during short rotations in obstetric medicine (OM). A Canadian research group created online clinical cases, the CanCOM cases, to overcome this. Methods: We conducted an exploratory study to document the use and perceived utility of the CanCOM cases. 77 residents doing an OM rotation participated in our study. We used a survey to document their perception of CanCOM cases (12 items, 7-point scale), clinical exposure to several conditions (pre and post rotation; 41 items, 7-point scale) and use of the educational tool (1 item, 4-option scale). Results: CanCOM cases was perceived as an accessible and useful tool. Participants completed a median of 6/20 cases (range 1-20), and highly recommended the cases (6.48 ± 0.73 SD on a 7-point Likert scale). Conclusion: Despite some technical limitations, CanCOM cases was shown to contribute to clinical exposure to rare but essential medical conditions.

Eleven ways to get a grip on the implementation of remote administration of high-stakes assessments.

The COVID-19 pandemic rushed licensure and certification institutions, as well as many university programs, to integrate Information and Communication Technologies (ICTs) in their practices to allow for remote administrations of their exams independent of distancing measures. The Black Ice covered in this manuscript is the integration of ICTs to allow remote administration of high-stakes assessments in terms of its development, administration, and monitoring with the aim to promote the validity of score interpretation.

La pandémie de la COVID-19 a eu comme conséquence de rendre nécessaire le recours aux technologies de l'information et de la communication aux évaluations des ordre professionnels, institutions de certification ainsi que de nombreux programmes universitaires. Le terrain glissant exploré dans cet article est celui de l'utilisation des TIC pour permettre la réalisation d'évaluations à enjeux élevés à distance par rapport à leur élaboration, l'administration et le monitorage tout en assurant la validité de l'interprétation des scores.

Validity as a social imperative: users' and leaders' perceptions.

Introduction: Recently, validity as a social imperative was proposed as an emerging conceptualization of validity in the assessment literature in health professions education (HPE). To further develop our understanding, we explored the perceived acceptability and anticipated feasibility of validity as a social imperative with users and leaders engaged with assessment in HPE in Canada. Methods: We conducted a qualitative interpretive description study. Purposeful and snowball sampling were used to recruit participants for semi-structured individual interviews and focus groups. Each transcript was analyzed by two team members and discussed with the team until consensus was reached. Results: We conducted five focus group and eleven interviews with two different stakeholder groups (users and leaders). Our findings suggest that the participants perceived the concept of validity as a social imperative as acceptable. Regardless of group, participants shared similar considerations regarding: the limits of traditional validity models, the concept's timeliness and relevance, the need to clarify some terms used to characterize the concept, the similarities with modern theories of validity, and the anticipated challenges in applying the concept in practice. In addition, participants discussed some limits with current approaches to validity in the context of workplace-based and programmatic assessment. Conclusion: Validity as a social imperative can be interwoven throughout existing theories of validity and may represent how HPE is adapting traditional models of validity in order to respond to the complexity of assessment in HPE; however, challenges likely remain in operationalizing the concept prior to its implementation.

Contexte: Une nouvelle manière de concevoir la validité en matière d'évaluation des apprentissages dans les programmes en sciences de la santé a récemment été proposée : la validité en tant qu'impératif social. Pour mieux la comprendre, nous avons exploré l'acceptabilité perçue et la faisabilité anticipée de la validité en tant qu'impératif social auprès d'utilisateurs et de leaders en matière d'évaluation en éducation des professions de la santé au Canada. Méthodes: Nous avons mené une étude qualitative descriptive interprétative. Pour recruter les participants aux entrevues individuelles semi-structurées et aux groupes de discussion, nous avons utilisé l'échantillonnage par choix raisonné et l'échantillonnage boule de neige. Les transcriptions ont été analysées par deux membres de l'équipe et discutées avec l'ensemble de l'équipe jusqu'à l'obtention d'un consensus. Résultats: Nous avons mené cinq groupes de discussion et onze entretiens avec deux groupes de parties prenantes, l'un composé d'utilisateurs, l'autre de leaders. Nos résultats suggèrent que les participants estiment acceptable le concept de validité comme impératif social. Quel que soit le groupe, les participants ont partagé des considérations similaires concernant : les limites des modèles de validité traditionnels, l'actualité et la pertinence du concept, la nécessité de clarifier certains termes utilisés pour définir le concept, les similitudes avec les théories modernes de la validité, et les défis anticipés de son application. En outre, les participants ont soulevé certaines limites des approches actuelles de la validité dans le contexte de l'évaluation en milieu de travail et de l'évaluation programmatique. Conclusion: La notion de validité comme impératif social peut être incorporée dans les théories existantes pour traduire l'adaptation des modèles traditionnels de la validité à la complexité de l'évaluation en éducation des professions de la santé; cependant, certains défis liés à l'opérationnalisation du concept seraient à résoudre avant sa mise en œuvre.

Effect of station format on the psychometric properties of Multiple Mini Interviews.

BACKGROUND: Given the widespread use of Multiple Mini Interviews (MMIs), their impact on the selection of candidates and the considerable resources invested in preparing and administering them, it is essential to ensure their quality. Given the variety of station formats used and the degree to which that factor resides in the control of training programmes that we know so little about, format's effect on MMI quality is a considerable oversight. This study assessed the effect of two popular station formats (interview vs. role-play) on the psychometric properties of MMIs. METHODS: We analysed candidate data from the first 8 years of the Integrated French MMIs (IF-MMI) (2010-2017, n = 11 761 applicants), an MMI organised yearly by three francophone universities and administered at four testing sites located in two Canadian provinces. There were 84 role-play and 96 interview stations administered, totalling 180 stations. Mixed design analyses of variance (ANOVAs) were used to test the effect of station format on candidates' scores and stations' discrimination. Cronbach's alpha coefficients for interview and role-play stations were also compared. Predictive validity of both station formats was estimated with a mixed multiple linear regression model testing the relation between interview and role-play scores with average clerkship performance for those who gained entry to medical school (n = 462). RESULTS: Role-play stations (M = 20.67, standard deviation [SD] = 3.38) had a slightly lower mean score than interview stations (M = 21.36, SD = 3.08), p < 0.01, Cohen's d = 0.2. The correlation between role-play and interview stations scores was r = 0.5 (p < 0.01). Discrimination coefficients, Cronbach's alpha and predictive validity statistics did not vary by station format. CONCLUSION: Interview and role-play stations have comparable psychometric properties, suggesting format to be interchangeable. Programmes should select station format based on match to the personal qualities for which they are trying to select.

Stakeholders' perception on the implementation of Developmental Progress Assessment: using the Theoretical Domains Framework to document behavioral determinants.

BACKGROUND: The widespread implementation of longitudinal assessment (LA) to document trainees' progression to independent practice rests more on speculative rather than evidence-based benefits. We aimed to document stakeholders' knowledge of- and attitudes towards LA, and identify how the supports and barriers can help or hinder the uptake and sustainable use of LA. METHODS: We interviewed representatives from four stakeholder groups involved in LA. The interview protocols were based on the Theoretical Domains Framework (TDF), which contains a total of 14 behaviour change determinants. Two team members coded the interviews deductively to the TDF, with a third resolving differences in coding. The qualitative data analysis was completed with iterative consultations and discussions with team members until consensus was achieved. Saliency analysis was used to identify dominant domains. RESULTS: Forty-one individuals participated in the study. Three dominant domains were identified. Participants perceive that LA has more positive than negative consequences and requires substantial ressources. All the elements and characteristics of LA are present in our data, with differences between stakeholders. CONCLUSION: Going forward, we could develop and implement tailored and theory driven interventions to promote a shared understanding of LA, and maintain potential positive outcomes while reducing negative ones. Furthermore, ressources to support LA implementation need to be addressed to facilitate its uptake.

Narrative Assessments in Higher Education: A Scoping Review to Identify Evidence-Based Quality Indicators.

PURPOSE: Narrative comments are increasingly used in assessment to document trainees' performance and to make important decisions about academic progress. However, little is known about how to document the quality of narrative comments, since traditional psychometric analysis cannot be applied. The authors aimed to generate a list of quality indicators for narrative comments, to identify recommendations for writing high-quality narrative comments, and to document factors that influence the quality of narrative comments used in assessments in higher education. METHOD: The authors conducted a scoping review according to Arksey & O'Malley's framework. The search strategy yielded 690 articles from 6 databases. Team members screened abstracts for inclusion and exclusion, then extracted numerical and qualitative data based on predetermined categories. Numerical data were used for descriptive analysis. The authors completed the thematic analysis of qualitative data with iterative discussions until they achieved consensus for the interpretation of the results. RESULTS: After the full-text review of 213 selected articles, 47 were included. Through the thematic analysis, the authors identified 7 quality indicators, 12 recommendations for writing quality narratives, and 3 factors that influence the quality of narrative comments used in assessment. The 7 quality indicators are (1) describes performance with a focus on particular elements (attitudes, knowledge, skills); (2) provides a balanced message between positive elements and elements needing improvement; (3) provides recommendations to learners on how to improve their performance; (4) compares the observed performance with an expected standard of performance; (5) provides justification for the mark/score given; (6) uses language that is clear and easily understood; and (7) uses a nonjudgmental style. CONCLUSIONS: Assessors can use these quality indicators and recommendations to write high-quality narrative comments, thus reinforcing the appropriate documentation of trainees' performance, facilitating solid decision making about trainees' progression, and enhancing the impact of narrative feedback for both learners and programs.

Drivers of iPad use by undergraduate medical students: the Technology Acceptance Model perspective.

BACKGROUND: Medical students need to acquire a continuously growing body of knowledge during their training and throughout their practice. Medical training programs should aim to provide students with the skills to manage this knowledge. Mobile technology, for example, could be a strategy used through training and practice. The objective of this study was to identify drivers of using mobile technology (an iPad) in a UGME preclinical settings and to study the evolution of those drivers over time. METHODS: We solicited all students from two cohorts of a preclinical component of a Canadian UGME program. They were asked to answer two online surveys: one on their first year of study and another on the second year. Surveys were built based on the Technology Acceptance Model (TAM) to which other factors were also added. Data from the two cohorts were combined and analysed with partial least squares structural equation modelling (PLS-SEM) to test two measurement models, one for each year. RESULTS: We tested fifteen hypotheses on both data sets (first year and second year). Factors that explained the use of an iPad the first year were knowledge, preferences, perceived usefulness and anticipation. In the second year, perceived usefulness, knowledge and satisfaction explained the use of an iPad. Other factors have also significantly, but indirectly influenced the use of the iPad. CONCLUSIONS: We identified factors that influenced the use of an iPad in a preclinical medical program. These factors differed from the first year to the second year in the program. Our results suggest that interventions should be tailored for different point in time to foster the use of an iPad. Further study should investigate how interventions based on these factors may influence implementation of mobile technology to help students acquire ability to navigate efficiently through medical knowledge.